Data Recorders of Autonomous Vehicles

ABSTRACT

Systems, methods and apparatus to collect sensor data generated in an autonomous vehicle. Sensors of the vehicle generate a sensor data stream that is buffered, in parallel and in a cyclic way, in a first cyclic buffer and a larger second cyclic buffer respectively. An advanced driver assistance system of the vehicle generates an accident signal when detecting or predicting an accident and provides a training signal when detecting a fault in object detection, recognition, identification or classification. The accident signal causes a sensor data stream segment to be copied from the first cyclic buffer into a slot of a non-volatile memory, selected from a plurality of slots in a round robin way. The training signal causes a sensor data stream segment to be copied from the second cyclic buffer into an area of the non-volatile memory outside of the slots reserved for the first cyclic buffer.

RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/263,359 filed Jan. 31, 2019, the entiredisclosures of which application are hereby incorporated herein byreference.

This application is related to U.S. patent application Ser. No.16/263,403, filed Jan. 31, 2019 and entitled “Autonomous Vehicle DataRecorders,” U.S. patent application Ser. No. 15/923,820, filed Mar. 16,2018 and issued as U.S. Pat. No. 10,846,955 on Nov. 24, 2020, U.S.patent application Ser. No. 15/938,504, filed Mar. 28, 2018 and entitled“Black Box Data Recorder with Artificial Intelligence Processor inAutonomous Driving Vehicle,” U.S. patent application Ser. No.16/010,646, filed Jun. 18, 2018 and issued as U.S. Pat. No. 11,094,148on Aug. 17, 2021, U.S. patent application Ser. No. 15/858,143, filedDec. 29, 2017 and entitled “Distributed Architecture for EnhancingArtificial Neural Network,” and U.S. patent application Ser. No.15/858,505, filed Dec. 29, 2017 and entitled “Self-Learning inDistributed Architecture for Enhancing Artificial Neural Network,” theentire contents of which applications are incorporated herein byreference.

TECHNICAL FIELD

At least some embodiments disclosed herein relate to recording sensordata of autonomous vehicles for subsequent analyses, such as accidentreviews and/or updating data, maps and artificial neural network modelsused by advanced driver assistance systems (ADAS).

BACKGROUND

Autonomous vehicles typically include many sensors to assist incontrolling the operations of the autonomous vehicles. In the case of anaccident, collision, or near collision involving an autonomous vehicle,there may be a benefit from reviewing the sensor data recorded justprior to and/or during the accident to assist in potentially determiningthe cause of the accident and/or whether there may have been a designflaw and/or a vehicle failure.

In the event of a power loss during the accident, vehicle sensor datastored in a volatile memory may be lost. A volatile memory requirespower to maintain data stored therein. Examples of volatile memoryinclude Dynamic Random-Access Memory (DRAM) and Static Random-AccessMemory (SRAM).

Some memory integrated circuits are non-volatile and can retain storeddata even when not powered for a long period of time (e.g., days,months, years). Examples of non-volatile memory include flash memory,Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), ErasableProgrammable Read-Only Memory (EPROM) and Electronically ErasableProgrammable Read-Only Memory (EEPROM) memory, etc.

Some non-volatile memories have useful service periods limited by thecycles of programming and erasing to store new data. A program erase(P/E) budget represents a predetermined number of cycles of programmingand erasing that can be performed reliably for replacing data in anerasable medium. After the predetermined number of cycles of erasure,the program erase (P/E) budget of such the erasable medium is used up;and as a result, the medium may become unreliable in a statistical senseand thus is considered at the end of its useful service life. Forexample, a single-level cell (SLC) flash memory can have a P/E budgetnear 100,000 cycles; a multi-level cell (MLC) flash memory can have aP/E budget ranging from 10,000 to 30,000 cycles; and a triple-level cell(TLC) or quad-level cell (QLC) flash memory can have a P/E budgetbetween 3,000 to 5,000 cycles.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 shows a system in which a vehicle is configured with a datarecorder to collect sensor data for improving its advanced driverassistance systems (ADAS) and/or for accident review.

FIG. 2 shows an autonomous vehicle having a data recorder according toone embodiment.

FIG. 3 shows a data recorder according to one embodiment.

FIG. 4 shows a method of operating a data recorder according to oneembodiment.

FIG. 5 shows another method of operating a data recorder according toone embodiment.

DETAILED DESCRIPTION

At least some embodiments disclosed herein provide systems, methods andapparatus to collect and store sensor data generated in an autonomousvehicle, or another vehicle with an advanced driver assistance system(ADAS), under different conditions and for different purposes.

In one example, after an accident (e.g., collision or near-collision) ora near accident event, the sensor data can be reviewed to determine thecause of the event. Such sensor data can be herein referred to asaccident sensor data. The accident sensor data can be analyzed to, forexample, identify the cause of the accident or near accident eventand/or unsafe aspects or design of the autonomous vehicle. The analysiscan lead to improved control designs and/or configurations for the safeoperations of the autonomous vehicle and/or similar vehicles.

In another example, after the ADAS (e.g., autonomous driving system) ofthe vehicle detects a fault/error in processing the sensor data, or amismatch in identifying an object from the sensor data, or a failure toidentify/classify an object from the sensor data, or a failure torecognize, detect, identify or classify an object with a minimumconfidence level, the sensor data can be used to improve the advanceddriver assistance system (ADAS). Such sensor data not associated with anaccident or a near accident event can be herein referred to as trainingsensor data. The improvement can be made, for example, throughperforming machine learning using the sensor data and/or updating thedata model(s) used in the ADAS to classify or identify an object. Suchdata models can include a high definition (HD) map of a road system inwhich the autonomous vehicle travels and/or an artificial neural networkused for object detection, classification, recognition, and/oridentification.

A high definition (HD) map is typically used in an autonomous vehiclefor motion planning. Such a map can have high precision at centimeterlevel. The HD map can provide geometric and semantic information aboutthe environment (e.g., a road system) in which the vehicle operates.Geometric information provided in the HD map can include raw sensor datacollected by light detection and ranging (lidar), radar, camera, sonar,GPS, etc. Semantic information can identify objects, such as laneboundaries, intersections, parking spots, stop signs, traffic lights,and other information, such as traffic speeds, lane change restrictions,etc.

Some techniques have been developed to further use the HD map togenerate representations/inputs for object detection, classification,identification and/or recognition performed using an artificial neuralnetwork (ANN).

Objects identified in the HD map can be compared with objects identifiedvia sensor data generated by digital cameras, lidar (light detection andranging), radar (radio detection and ranging), ultrasound sonar (soundnavigation and ranging), etc. When there is a mismatch in objectdetection, classification, recognition, or identification, the sensordata can be used in an analysis to update the HD map and/or train theartificial neural network (ANN) used for object detection,classification, recognition, or identification.

In some embodiments disclosed herein, a data recorder of an autonomousvehicle is configured with two separate cyclic buffers for bufferingaccident sensor data for subsequent accident review and for bufferingtraining sensor data for improving ADAS (e.g., an HD map and/or an ANNmodel). The separate cyclic buffers can be separately configured to meetthe different requirements for buffering sensor data for differentusages (e.g., accident review and ADAS improvement). In otherembodiments disclosed herein, a single cyclic buffer is controlled by acontroller to meet the different requirements for the two separatecyclic buffers.

For example, the data recorder can be configured with an accident datacyclic buffer to buffer accident sensor data for subsequent accidentreview and a training data cyclic buffer to buffer training sensor datafor improving ADAS.

For example, the data recorder can receive a data stream containing livevideo data from various cameras mounted on the vehicle and variousoutput signals from the other sensors, such as radar, lidar, ultrasoundsonar, etc. The sensor data can be raw data from the sensors orcompressed image/video data.

Each of the cyclic buffers is configured to buffer data cyclically,where incoming data is filled into the buffer until the buffer is full,and then further new data is buffered to replace the oldest data in thecyclic buffer. The cyclic buffer can be implemented using a type ofmemory that has very high endurance (e.g., allowing hundreds ofpetabytes of data to be written and rewritten in the cyclic buffer). Forexample, the cyclic buffer can be implemented via SRAM or DRAM that arevolatile and require power to retain its content. Alternatively, thecyclic buffer can be implemented using a non-volatile memory that hasvery high endurance, such as a cross point memory.

A cross point memory uses transistor-less memory elements, each of whichhas a memory cell and a selector that are stacked together as a column.Memory element columns are connected via two perpendicular layers ofwires, where one layer is above the memory element columns and the otherlayer below the memory element columns. Each memory element can beindividually selected at a cross point of one wire on each of the twolayers. Cross point memory devices are fast and non-volatile and can beused as a unified memory pool for processing and storage.

Some non-volatile memory that has very high endurance suffers from shortdata retention (e.g., minutes or hours).

The autonomous vehicle can be configured to generate a signal indicatingthe occurrence of an accident or a prediction of a potential accident.Such a signal can be herein referred to as an accident signal.

An accident signal can be generated by the ADAS (e.g., autonomousdriving system) based on sensor data. For example, an activation of theautomatic emergency braking can cause the generation of an accidentsignal for the data recorder. For example, an activation of theanti-lock braking system can cause the generation of an accident signalto the data recorder. For example, accelerating faster than a thresholdcan cause the generation of an accident signal to the data recorder. Forexample, when a collision avoidance system of the vehicle detects animpact threat, or determines that the probability of an impact is abovea threshold, an accident signal can be generated for the data recorder.For example, when airbag sensors determine that a collision is occurringduring the initial impact of a crash, an accident signal can begenerated for the data recorder. For example, when the ADAS detects fromthe sensor data a dangerous scenario, such as another car or object iscloser than a threshold to the autonomous vehicle, an accident signal isgenerated for the data recorder.

The data recorder of the vehicle is configured to have a non-volatilememory. The non-volatile memory of the data recorder can be implementedusing a high density and high data retention flash memory that canretain data for months and/or years without being powered. For example,a triple-level cell (TLC), or quad-level cell (QLC) NAND flash memorycan be used as the non-volatile memory of the data recorder.

In response to the accident signal, the data recorder copies the contentof the accident data cyclic buffer into the non-volatile memory suchthat the accident sensor data can be preserved after the accident, whichmay prevent the data recorder from being powered for a long period oftime.

Preferably, the non-volatile memory of the data recorder is configuredto include a number of slots for accident sensor data. Each of the slotsis a dedicated storage space for saving the sensor data copied from theaccident data cyclic buffer. In response to each accident signal, thecontent of the accident data cyclic buffer is copied into an availableone of the slots in the non-volatile memory. Initially, all of the slotsare available for storing sensor data copied from the accident datacyclic buffer. After all the slots have been used in response to pastaccident signals, the slot storing the sensor data associated with theoldest accident signal is made available to store the new, currentsensor data copied from the accident data cyclic buffer in response tothe current accident signal.

The data recorder is configured to ensure the completion of the copyingof sensor data from the accident data cyclic buffer to the non-volatilememory, even when the accident causes disruption in power supply to thedata recorder. For example, the data recorder can have a backup powersource that is sufficient to power the operations to complete thecopying of sensor data from the accident data cyclic buffer to thenon-volatile memory, even when the external power supply to the datarecorder is removed immediately after the accident signal. The backuppower source can be implemented via a capacitor that is charged to storepower for the operations while the data recorder is powered normally viaan external power supply (e.g., during the operation of recording datain the accident data cyclic buffer before the accident). Alternatively,or in combination, the backup power source can include a battery mountedwithin the housing of the data recorder.

The accident data cyclic buffer is configured to buffer sensor data fora period of time (e.g., 30 seconds) immediately before the accidentsignal. In response to the accident signal, the sensor data buffered forthis period of time is copied into a slot in the non-volatile memory ofthe data recorder; and before the respective data is copied into theslot, the existing data in the accident data cyclic buffer at the timeof the accident signal is not overwritten with new data from thesensors.

In one implementation, in response to the accident signal, the accidentdata cyclic buffer temporarily stops buffering incoming new sensor datato preserve its content and resumes buffering data after at least anoldest portion of the content in the accident data cyclic buffer hasbeen copied into the slot in the non-volatile memory of the datarecorder. Optionally, the accident data cyclic buffer can resumebuffering of incoming new sensor data after the completion of storingthe existing data for the accident signal into the non-volatile memory.Optionally, the accident data cyclic buffer can resume buffering ofincoming new sensor data after a predetermined portion of the content ofthe accident data cyclic buffer has been copied into a slot in thenon-volatile memory and before the entire content of the accident datacyclic buffer is copied into the slot, where buffering of incoming newsensor data does not write over the accident sensor data to be copied tothe non-volatile memory of the data recorder.

In another implementation, in response to the accident signal, theaccident data cyclic buffer is configured to read an oldest portion ofthe content in the accident data cyclic buffer for recording in thenon-volatile memory and then record the incoming new sensor data inplace to overwrite the oldest portion in the accident data cyclicbuffer. Thus, the data recorder vacates the content from the accidentdata cyclic buffer into a slot in the non-volatile memory whilecontinuing recording the incoming sensor data in the portion of thevacated portion of the accident data cyclic buffer; and the datarecorder can continue buffering incoming sensor data, without failing tobuffer any incoming sensor signal, while copying the content associatedwith the accident signal (e.g., generated during the 30 second timeperiod immediately before the accident signal) into a slot in thenon-volatile memory.

Optionally, the data recorder and/or the autonomous vehicle can beconfigured to upload the data content from the slots in the non-volatilememory to a remote server through a communication network/connection.For example, the data transfer can be configured to perform in one ofseveral options. One option is to wirelessly transmit the data via acellular communications network to the server. Another option is towirelessly transmit the data to the server via a wireless local areanetwork (e.g., Wi-Fi). A further option is to transmit through a wiredconnection over a diagnosis port of the autonomous vehicle. A yetanother option is to remove the data recorder from the autonomousvehicle for a wired connection to a communication port of the datarecorder.

The accident sensor data recorded in the slots of the non-volatilememory of the data recorder can be used in the investigation of theaccidents or near accident events.

The autonomous vehicle can generate a signal indicating the request tosave sensor data for training and/or improving ADAS. Such a signal canbe herein referred to as a training signal.

For example, the training signal can be generated by the ADAS (e.g.,autonomous driving system) in response to a mismatch between roadobjects identified in the HD map (e.g., lane boundaries, obstacles,traffic signs, etc.) and the corresponding objects detected orrecognized from the sensor data. The sensor data associated with themismatch can be used to train the ADAS in recognizing the objects and/orupdating the HD map for the corresponding segment of the road.

The training data cyclic buffer is configured to buffer training sensordata in parallel with the accident data cyclic buffer buffering accidentsensor data. The training data cyclic buffer is configured to have acapacity to store data collected in a longer period of time (e.g., 3minutes) than the accident data cyclic buffer (e.g., 30 seconds).

Responsive to a training signal, training sensor data is copied from thetraining data cyclic buffer into the non-volatile memory of the datarecorder. The training sensor data being copied responsive to thetraining signal can include a portion that is generated prior to thetraining signal and a remaining portion that is generated after thetraining signal. In some instances, the portion generated prior to thetraining signal is for a time period equal to or longer than the lengthof the stream segment buffered in the accident data cyclic buffer (e.g.,30 seconds or more). In other instances, the entire training sensor databeing copied responsive to the training signal is generated before thetraining signal. Optionally, a time period selection indication isprovided with the training signal, such that the data recorder canselectively copy the portion generated prior to the training signal andthe remaining portion generated after the training signal.Alternatively, the ADAS can select the time period by adjusting thetiming of the training signal such that a time window predefinedrelative to the training signal is the time period of the sensor datastream segment that is to be saved as the training sensor data. In someinstances, upon receiving the training signal in the data recorder, thetraining data cyclic buffer continues buffering incoming sensor datasignal until the remaining portion, generated after the training signal,is buffered in the training data cyclic buffer for copying into thenon-volatile memory of the data recorder. Optionally, the training datacyclic buffer may temporarily stop buffering further incoming sensordata after the remaining portion that is generated after the trainingsignal is buffered in the training data cyclic buffer and before thecompletion of copying the training sensor data associated with thetraining signal to the non-volatile memory of the data recorder. Thecopying of the training sensor data can be configured to start inresponse to the training signal and before the completion of thebuffering the entire training sensor data associated with the trainingsignal. Alternatively, the copying of the training sensor data can beconfigured to start upon or after the completion of the buffering theentire training sensor data associated with the training signal.

Preferably, the non-volatile memory of the data recorder is configuredto have a dedicated area reserved to store training sensor data. Thetraining data area in the non-volatile memory is separate from the slotsconfigured for sensor data for accident reviews.

Optionally, the training data area can have multiple slots for storingtraining sensor data associated with multiple training signals. The datarecorder can be configured to retain the training sensor data associatedwith the latest training signals.

Optionally, the data recorder is further configured to receive priorityindication of the training signals. For example, the ADAS can evaluate adegree of mismatch between the object identified in the HD map and thecorresponding object detected/recognized from the sensor data. A highdegree of mismatch is assigned a high priority. Similarly, a lowconfidence level in detecting, classifying, recognizing or identifyingan object can be assigned a high priority. Thus, the data recorder canbe configured to retain the training sensor data based on the priorityindications when the data recorder has insufficient slots to storetraining data that have not yet been transmitted to a server.

The content copied into the training data area of the non-volatilememory of the data recorder can be transmitted automatically to aremoter server via some of the communication options discussed above forthe transmission of accident data, such as a wireless cellularcommunication network (e.g., 5G cellular communication) and/or awireless local area network (e.g., Wi-Fi). Depending on the networkbandwidth and coverage, the data transmission may take a time periodfrom minutes to hours/days. The non-volatile memory of the data recordercan retain the data without power before the completion of the datatransmission.

FIG. 1 shows a system in which a vehicle is configured with a datarecorder to collect sensor data for improving its advanced driverassistance systems (ADAS) and for accident review.

The system of FIG. 1 includes a vehicle (111) having a data recorder(101). The data recorder (101) can be configured according to any of theembodiments disclosed herein. The vehicle (111) has an advanced driverassistance system (ADAS) (105) and one or more sensors (103) thatprovide sensor data input to the ADAS (105).

For example, the sensor(s) (103) can include digital cameras, lidars,radars, ultrasound sonars, brake sensors, speed sensors, accelerationsensors, airbag sensors, a GPS (global positioning system) receiver,etc.

The outputs of the sensor(s) (103) as a function of time are provided asa sensor data stream to the ADAS (105) to support its operations and tothe data recorder (101) for buffering and/or recording. The ADAS (105)can generate accident signals and/or training signals to initiate thetransfer of sensor data from the cyclic buffers to the non-volatilememory of the data recorder (101), as discussed above.

For example, in response to the detection of an accident or a nearaccident event, the ADAS (105) can generate an accident signal to causethe data recorder (101) to store accident sensor data (e.g., 127) of atime duration B (e.g., 30 seconds). The accident sensor data (127) istypically generated by the sensor(s) in the time duration B (e.g., 30seconds) and buffered in a cyclic buffer of the data recorder (101)prior to the accident signal generated by the ADAS (105).

For example, in response to the detection of an opportunity to train orimprove the ADAS (105) that is other than an accident or a near accidentevent, the ADAS (105) can generate a training signal to cause the datarecorder (101) to store training sensor data (e.g., 121) of a timeduration A (e.g., 3 minutes). The training sensor data (121) isgenerated by the sensor(s) in the time duration A (e.g., 3 minutes) andbuffered in a cyclic buffer of the data recorder (101). The timeduration A (e.g., 3 minutes) can include a portion that is before atraining signal generated by the ADAS (105) and optionally a portionthat is after the training signal.

The vehicle (111) is configured with a wireless communication device totransmit, via wireless signals (113) and a communication network (117)to a remote server (119), the accident sensor data (127) and/or thetraining sensor data (121). The remote server (119) is typicallyconfigured at a location away from a road (102) on which the vehicle(111) is in service. One example of the communication network (117) is acell phone network having one or more base stations (e.g., 115) toreceive the wireless signals (e.g., 113). Another example of thecommunication network (117) is internet, where the wireless local areanetwork signals (e.g., 113) transmitted by the vehicle (111) is receivedin an access point (e.g., 115) for further communication to the server(119). In some implementations, the vehicle (111) uses a communicationlink (107) to a satellite (109) or a communication balloon to transmitthe sensor data (e.g., 121 or 127) to the server (119).

The server (119) can be configured to include an accident module (129).The accident sensor data (127) can be reviewed and/or analyzed in theaccident module (129) to identify the cause of the accident and/orpossible design or configuration changes to improve autonomous drivingand/or the ADAS.

The server (119) can be configured to include a training and updatingmodule (123). The training sensor data (121) can be used in the trainingand updating module (123) to improve the artificial neural network (ANN)(125) of the ADAS (105) and/or to update the high definition (HD) map(126) of the road (102) used by the vehicle (111).

In some implementations, the ADAS (105) can optionally include a machinelearning module that is configured to use the training sensor data (121)stored in the non-volatile memory of the data recorder (101) to improveits ANN, with or without assistance from the server (119).

FIG. 2 shows an autonomous vehicle (111) having a data recorder (101)according to one embodiment. For example, the vehicle (111) in thesystem of FIG. 1 can be implemented using the autonomous vehicle (111)of FIG. 2.

The vehicle (111) of FIG. 2 is configured to have an advanced driverassistance system (ADAS) (105). The ADAS (105) of the vehicle (111) canhave a high definition (HD) map (126) for motion planning and anArtificial Neural Network (ANN) (125) for object detection, recognition,identification, and/or classification. Optionally, the HD map (126) canalso be used in object detection, recognition, identification, and/orclassification.

When there is a mismatch between an object identified in the HD map(126) and the identification of the object via the ANN (125), the ADAS(105) can generate a training signal for the data recorder (101).

When the ADAS (105) detects an accident or a near accident event, theADAS (105) can generate an accident signal for the data recorder (101).

The vehicle (111) typically includes an infotainment system (149), acommunication device (139), one or more sensors (103), and a computersystem (131) that is connected to some controls of the vehicle (111),such as a steering control (141) for the direction of the vehicle (111),a braking control (143) for stopping of the vehicle (111), anacceleration control (145) for the speed of the vehicle (111), etc. Insome embodiments, the vehicle (111) in the system of FIG. 1 has asimilar configuration and/or similar components.

The computer system (131) of the vehicle (111) includes one or moreprocessors (133), a data recorder (101), and memory (135) storingfirmware (or software) (147), including the computer instructions anddata models for ADAS (105). The data models of the ADAS (105) caninclude the HD map (126) and the ANN (125).

The one or more sensors (103) can include a visible light camera, aninfrared camera, a lidar, radar, or sonar system, a peripheral sensor, aglobal positioning system (GPS) receiver, a satellite positioning systemreceiver, a brake sensor, and/or an airbag sensor. The sensor(s) canprovide a stream of real time sensor data to the computer system (131).The sensor data generated by a sensor (103) of the vehicle can includean image that captures an object using a camera that images using lightsvisible to human eyes, or a camera that images using infrared lights, ora sonar, radar, or LIDAR system. Image data obtained from at least onesensor of the vehicle is part of the collected sensor data for recordingin the data recorder.

The outputs of the ADAS (105) can be used to control (e.g., (141),(143), (145)) the acceleration of the vehicle (111), the speed of thevehicle (111), and/or the direction of the vehicle (111), duringautonomous driving.

In some embodiments, after the server (119) updates the ANN (125) and/orthe HD map (126), the server (119) can update the ADAS of the vehicle(111) over the air by downloading the ANN (125) and/or the HD map (126)to the computer system (131) of the vehicle (111) through thecommunication device (139) and/or wireless signals (113).

FIG. 3 shows a data recorder according to one embodiment. For example,the data recorder (101) in the system of FIG. 1 and/or in the vehicle(111) of FIG. 2 can be implemented using the data recorder (101) of FIG.3.

The data recorder (101) of FIG. 3 can include a housing (137) thatencloses its components. Optionally, the housing (137) of the datarecorder (101) seals the components of the data recorder (101) andprotects the data recorder (101) and/or the data stored in thenon-volatile memory (165) from destruction during an accident; and thus,the data stored in the non-volatile memory (165) of the data recorder(101) can be retrieved after an impact/collision, a fire, and/or a floodthat is typical in an accident involving a vehicle.

The data recorder (101) of FIG. 3 includes one or more communicationinterfaces or devices (153), an accident data cyclic buffer (161), atraining data cyclic buffer (163), a controller (159), and anon-volatile memory (165).

The communication interface(s) or device(s) (153) can be used to receivea sensor data stream (151) from the sensor(s) (103) of the vehicle (111)in which the data recorder (101) is installed. The sensor data stream(151) is provided to the accident data cyclic buffer (161) and thetraining data cyclic buffer (163) in parallel.

Further, communication interface(s) or device(s) (153) can be used toreceive training signals (e.g., 157) and accident signals (155) from thevehicle (111) and/or its ADAS (105).

In response to an accident signal (155), the controller (159) stopsrecording the incoming sensor data stream (151) into the accident datacyclic buffer (161) and starts copying the content from the accidentdata cyclic buffer (161) to the non-volatile memory (165).

The non-volatile memory (165) has multiple slots (171, 173, . . . ,179). Each of the slots is sufficient to store the entire content of theaccident data cyclic buffer (161). The controller (159) can use theslots (171, 173, . . . , 179) in a round robin way and/or in a cyclicway such that after a number of accident signals, the slots (171, 173, .. . , 179) stores the latest set of accident sensor data associated withand/or identified by the most recent accident signals.

In response to a training signal (157), the controller (159) cancontinue the buffering of the incoming sensor data stream (151) into thetraining data cyclic buffer (163) such that the training data cyclicbuffer (163) contains a training sensor data set that is partiallyrecorded before the training signal (157) and partially recorded afterthe training signal (157). The controller (159) then copies the trainingsensor data set into the area (167) reserved in the non-volatile memory(165) for training sensor data.

Optionally, the area (167) can also include multiple slots, each beingsufficient to store the content of the training data cyclic buffer(163).

Optionally, the data recorder (101) includes a backup power source, suchas a capacitor or a rechargeable battery.

FIG. 4 shows a method of operating a data recorder according to oneembodiment. For example, the method of FIG. 4 can be implemented in thedata recorder (101) of FIG. 3 in the vehicle (111) of FIG. 2 in thesystem of FIG. 1.

At blocks 201 and 211, a data recorder (101) provides a first cyclicbuffer (161) and a second cyclic buffer (163).

At block 203, the data recorder (101) buffers in the first cyclic buffer(161) a first sensor data stream of a first time duration (e.g., 30seconds).

At block 213, the data recorder (101) buffers in the second cyclicbuffer (163) a second sensor data stream of a second time duration(e.g., 3 minutes).

The first and second cyclic buffers (161 and 163) are operated inparallel. Data is buffered into each of the cyclic buffers (161 and 163)in a cyclic way such that when the buffer is full, the newest data iswritten over the oldest data. For example, data is recorded from thebeginning of the buffer towards the end of the buffer. Once reaching theend of the buffer, data is again writing from the beginning of thebuffer towards the end of the buffer, as if the end of the bufferfollows the beginning of the buffer.

At block 205, the data recorder (101) receives a request to recordsensor data. The request can be an accident signal or a training signal.

At block 207, the data recorder (101) determines whether the request isfor recording an accident (e.g., a collision or near collision).

If the request is for recording an accident, at block 209 the datarecorder (101) copies first content from the first cyclic buffer (161)to a non-volatile memory (165).

Optionally, the data recorder (101) stops, for a period of time, thebuffering of incoming sensor data into the first cyclic buffer (161), asa response to the request to record an accident (e.g., to preventoverwritten existing data in the first cyclic buffer (161) and/or topreserve backup power when the external power supply to the datarecorder (101) is interrupted).

At block 221, the data recorder (101) transmits or provides the firstcontent from the non-volatile memory (165) to a server (119) after theaccident.

At block 223, the server (119) reviews the accident based on the firstcontent.

If the request is not recording an accident, at block 219 the datarecorder (101) copies second content from the second cyclic buffer (163)to the non-volatile memory (165).

Optionally, the data recorder (101) continues, for a period of time, thebuffering of incoming sensor data into the first cyclic buffer (161), asa response to the request and thus buffers the second content containingsensor data for a time duration that is partially before the request andpartially after the request. After the period of continued buffering,the data recorder (101) may stop buffering for a further period of timebefore the completion of the copying of the second content into thenon-volatile memory (165).

At block 231, the data recorder (101) transmits the second content fromthe non-volatile memory (165) to a server (119).

At block 233, the server (119) improves an advanced driver assistantsystem (ADAS) based on the second content.

FIG. 5 shows another method of operating a data recorder according toone embodiment. For example, the method of FIG. 5 can be implemented inthe data recorder (101) of FIG. 3 in the vehicle (111) of FIG. 2 in thesystem of FIG. 1. The method of FIG. 5 can be performed in combinationwith the method of FIG. 4.

At block 251, a data recorder (101) provides a first cyclic buffer (161)for buffering sensor data related to an accident and a second cyclicbuffer (163) for buffering sensor data not related to accidents.

At block 253, a data recorder (101) provides a non-volatile memory(165).

At block 255, the data recorder (101) partitions the non-volatile memory(165) into a first portion (171, 173, . . . , 179) and a second portion(167), where the first portion has a storage capacity that is multipletimes of the capacity of the first cyclic buffer (161) and the secondportion is sufficient to store data buffered in the second cyclic buffer(163).

At block 257, the data recorder (101) organizes the first portion into apredetermined number of slots (171, 173, . . . , 179), each having acapacity to store the data buffered in the first cyclic buffer (161).

At block 259, the data recorder (101) stores sensor data for up to thepredetermined number of most recent accidents in the predeterminednumber of slots (171, 173, . . . , 179).

For example, the data recorder (101) copies the accident sensor datafrom the accident data cyclic buffer (161) into the first slot (171) inresponse to an initial accident signal. For each subsequent accidentsignal, the data recorder (101) store the accident sensor data in thenext slot (e.g., 173, . . . , 179). After reaching the last slot (179),the data recorder (101) uses the first slot (171) as the next slot, asif the first slot (171) is the next slot following the last slot (179).

Optionally, the second portion (167) reserved for the training sensordata buffered in the training data cyclic buffer (163) is alsoconfigured to have multiple slots. Each of the slot stores an indicatoridentifying the priority of the training sensor data stored in therespective slot. When the slots in the second portion (167) are full,the data recorder (101) reuses the slot having the lowest priority thatis lower than the current training sensor data in the training datacyclic buffer (163).

In one example, an autonomous vehicle (111) has sensors (103) configuredto generate a sensor data stream (151) during operations of theautonomous vehicle (111) on a road (102). An advanced driver assistancesystem (ADAS) (105) of the vehicle (111) can be configured to operatethe vehicle autonomously (111) on the road (102) based on the sensordata stream (151). The advanced driver assistance system (ADAS) (105) isfurther configured to generate an accident signal (155) in response todetection or prediction of an accident and generate a training signal(157) in response to a fault in object detection, recognition,identification or classification.

A data recorder (101) of the vehicle (111) in the example includes anon-volatile memory (165), a first cyclic buffer (161), a second cyclicbuffer (163), and a controller (159). The first cyclic buffer (161) hasa capacity to buffer a first segment of the sensor data stream (151)(e.g., 30 seconds); and the second cyclic buffer has a capacity tobuffer a second segment of the sensor data stream (151) (e.g., 3minutes). In absence of the accident signal and the training signal, thefirst cyclic buffer (161) and the second cyclic buffer (163) areconfigured to buffer the sensor data stream (151) in parallel andcyclically within their respective capacities respectively. Before thegeneration of the accident signal (155) and the training signal (157),the sensor data stream segment buffered in the second cyclic buffer(163) can include the sensor data stream segment buffered in the firstcyclic buffer (161).

In response to the accident signal (155), the controller (159) isconfigured to copy a sensor data stream segment from the first cyclicbuffer (161) into the non-volatile memory (165). In response to thetraining signal (157), the controller (159) is configured to copy asensor data stream segment from the second cyclic buffer (163) into thenon-volatile memory (165).

The sensor data stream (151) from the sensors (103) can include outputsof the sensors (103), such as a digital camera, a radar, a lidar, or anultrasound sonar, or any combination thereof. The advanced driverassistance system (ADAS) (105) can include a map (126) of the road (102)and an artificial neural network (125) for detecting, recognizing,identifying and/or classifying objects on the road (102) based on thesensor data stream (151). In response to a mismatch between theidentification of an object in the map (126) of the road (102) and theidentification of a corresponding object performed by applying thesensor data stream (151) in the artificial neural network (125), theadvanced driver assistance system (ADAS) (105) can generate the trainingsignal (155).

Preferably, after the training signal (157), the autonomous vehicle(111) is configured to automatically transmit the sensor data streamsegment, copied into the non-volatile memory (165) responsive to thetraining signal (157), to a remote server (119). The server (119) can beconfigured with a training and updating module (123) to update the map(126) of the road (102) according to the sensor data stream segmentreceived from the vehicle (111), and/or to update the artificial neuralnetwork (125) using a machine learning technique, such as a supervisedmachine learning technique.

In response to the accident signal (155), the segment of the sensor datastream (151), generated prior to the accident signal (155) and/orbuffered in the first cyclic buffer (161) at the time of the accidentsignal (155), is copied into the non-volatile memory (165). Optionally,the controller (159) can stop the first cyclic buffer (161) frombuffering an incoming segment of the sensor data stream (151), followingimmediately after the accident signal (155), to prevent the loss of aportion of the segment of the sensor data stream (151) that is in thefirst cyclic buffer (161) at the time of the accident signal (155),before at least a portion of the content is copied from the first cyclicbuffer (161) to the non-volatile memory (165).

Optionally, in response to the training signal (157), the segment of thesensor data stream (151), generated partially before the training signal(157) and partially after the training signal (157), is buffered in thesecond cyclic buffer (163) and copied into the non-volatile memory(165). For example, the controller (159) can continue to use the secondcyclic buffer (163) to buffer a segment of the sensor data stream (151)that follows immediately after the training signal (157) to complete thebuffering of the segment to be copied into the non-volatile memory(165). The controller (159) can then stop the second cyclic buffer (163)from buffering a further incoming segment of the sensor data stream(151), once the segment of the sensor data stream (151) to be copiedinto the non-volatile memory (165) is in the second cyclic buffer (163).

In some implementations, the accident data cyclic buffer (161) is notused. In response to an accident signal (155), the controller (159)copies a portion of the content of the training data cyclic buffer (163)corresponding to the segment that would be buffered in the accident datacyclic buffer (161) into a slot in the non-volatile memory (165). Inother implementations, the training cyclic buffer (163) is not used. Inresponse to a training signal (157), the controller (159) copies atleast a portion of the content of the accident data cyclic buffer (161)corresponding to the segment that would be buffered in the training datacyclic buffer (163) into a slot (173) in the non-volatile memory (165),while directly storing a segment of the sensor data stream (151)following the training signal (155) into the area (167) of thenon-volatile memory (165). A combination of the segment copied from theaccident data cyclic buffer (161) into a slot in the corresponding slot(e.g., 173) and the segment saved directly into the area (167) providesthe training sensor data corresponding to the training signal (157).

In some implementations, in response to the accident signal (155), thecontroller (159) may stop the operation of the training data cyclicbuffer (163) to preserve power for copying data from the accident datacyclic buffer (161) to the non-volatile memory (e.g., 165) (e.g., inresponse to a detection of loss of external power supply to the datarecorder (101)).

The first and second cyclic buffers (161 and 163) can be volatilememories (e.g., DRAM or SRAM); and the non-volatile memory (165) caninclude TLC/QLC NAND flash memory. The first and second cyclic buffers(161 and 163) may buffer a common segment of the sensor data stream(151) prior to receiving an accident signal (155) or a training signal(157).

Optionally, the controller (159) partitions the non-volatile memory(165) into a first area (171, 173, . . . , 179) and a second area (167).Further, the controller (159) organizes the first area (171, 173, . . ., 179) into a plurality of slots (171, 173, . . . , 179), where eachslot (e.g., 171) has a storage capacity sufficient to store the entirecontent of the first cyclic buffer (161). A sensor data stream segmentassociated with and/or identified by a training signal (157) is copiedinto the second area (167); and a sensor data stream segment associatedwith and/or identified by an accident signal (155) is copied into one ofthe slots (171, 173, . . . , 179) in the first area. The controller(159) can use the slots according to a round robin scheme to maintain,in the slots (171, 173, . . . , 179), up to a predetermined number ofaccident sensor data stream segments associated with and/or identifiedby the most recent accident signals generated by the advanced driverassistance system (ADAS) (105) of the autonomous vehicle (111).

Optionally, the controller (159) can also organize the second area (167)of the non-volatile memory (165) into slots, where each slot has astorage capacity sufficient to store the entire content of the secondcyclic buffer (163). The controller (159) can store (e.g., in thenon-volatile memory or another memory) priority indicators of datastored in the slots. In response to the training signal (157), thecontroller (159) selects a slot from the slots in the second area (167),based on the priority indicators of the existing data in the slots inthe second area (167) and a priority indicator associated with thetraining signal (157). The training sensor data is copied from thesecond cyclic buffer (163) into the selected slot in the second area(167).

The present disclosure includes methods and apparatuses which performthe methods described above, including data processing systems whichperform these methods, and computer readable media containinginstructions which when executed on data processing systems cause thesystems to perform these methods.

The server (119), the computer system (131), and/or, the data recorder(101) can each be implemented as one or more data processing systems.

A typical data processing system may include an inter-connect (e.g., busand system core logic), which interconnects a microprocessor(s) andmemory. The microprocessor is typically coupled to cache memory.

The inter-connect interconnects the microprocessor(s) and the memorytogether and also interconnects them to input/output (I/O) device(s) viaI/O controller(s). I/O devices may include a display device and/orperipheral devices, such as mice, keyboards, modems, network interfaces,printers, scanners, video cameras and other devices known in the art. Inone embodiment, when the data processing system is a server system, someof the I/O devices, such as printers, scanners, mice, and/or keyboards,are optional.

The inter-connect can include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controllers include a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatileRAM (Random Access Memory), and non-volatile memory, such as hard drive,flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In the present disclosure, some functions and operations are describedas being performed by or caused by software code to simplifydescription. However, such expressions are also used to specify that thefunctions result from execution of the code/instructions by a processor,such as a microprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited tonon-transitory, recordable and non-recordable type media such asvolatile and non-volatile memory devices, read only memory (ROM), randomaccess memory (RAM), flash memory devices, floppy and other removabledisks, magnetic disk storage media, optical storage media (e.g., CompactDisk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.),among others. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A device, comprising: an interface operable toreceive: a sensor data stream from sensors configured on a vehicleduring autonomous operations of the vehicle on a road; and signals of afirst type and a second type; a non-volatile memory; a first cyclicbuffer having a first capacity; a second cyclic buffer having a secondcapacity larger than the first capacity, wherein, in absence of a signalof the first type and a signal of the second type, the first cyclicbuffer and the second cyclic buffer are configured to buffer the sensordata stream in parallel and cyclically within the first capacity and thesecond capacity respectively; and a controller configured to: copy, inresponse to the signal of the first type, first sensor data from thefirst cyclic buffer into the non-volatile memory; and copy, in responseto the signal of the second type, second sensor data from the secondcyclic buffer into the non-volatile memory.
 2. The device of claim 1,wherein the first capacity is configured to buffer the first sensor datagenerated by the sensors up to a first period of time; and the secondcapacity is configured to buffer the second sensor data generated by thesensors up to a second period of time that is longer than the firstperiod of time.
 3. The device of claim 2, wherein an advanced driverassistance system is configured to operate the vehicle on the road basedon the sensor data stream.
 4. The device of claim 3, wherein the sensorsinclude a digital camera, a radar, a lidar, or an ultrasound sonar, orany combination thereof.
 5. The device of claim 3, wherein the signal ofthe first type is from the advanced driver assistance system to indicatean accident involving the vehicle; and the signal of the second type isfrom the advanced driver assistance system to indicate an opportunity totrain the advanced driver assistance system.
 6. The device of claim 5,wherein the advanced driver assistance system includes an artificialneural network and is configured to generate the signal of the secondtype in response to a mismatch between a first identification of anobject determined by the artificial neural network using the sensor datastream and a second identification of the object determined withoutusing the sensor data stream.
 7. The device of claim 6, furtherconfigured to transmit the second sensor data from the non-volatilememory to a remote server configured to update the artificial neuralnetwork.
 8. The device of claim 5, wherein the first sensor data isentirely generated by the sensors prior to the signal of the first type;and the second sensor data includes a first portion that is generated bythe sensors prior to the signal of the second type and a second portionthat is generated by the sensors during a predetermined period of timeafter the signal of the second type.
 9. The device of claim 8, whereinthe controller is further configured to: stop, in response to the signalof the first type, buffering data from the sensor data stream into thefirst cyclic buffer before at least a portion of the first sensor datais copied from the first cyclic buffer into the non-volatile memory; andcontinue, for the predetermined period of time and in response to thesignal of the second type, buffering data from the sensor data streaminto the second cyclic buffer before the controller stops bufferingfurther data from the sensor data stream into the second cyclic buffer.10. A method, comprising: receiving, in an interface of a device, asensor data stream from sensors configured on a vehicle duringautonomous operations of the vehicle on a road, the interface operableto receive signals of a first type and a second type; in absence of asignal of the first type and a signal of the second type: buffering, bya first cyclic buffer having a first capacity, the sensor data stream;and buffering, by a second cyclic buffer having a second capacity largerthan the first capacity, the sensor data stream in parallel with thefirst cyclic buffer; copying, by a controller of the device and inresponse to the signal of the first type, first sensor data from thefirst cyclic buffer into a non-volatile memory of the device; andcopying, by the controller of the device and in response to the signalof the second type, second sensor data from the second cyclic bufferinto the non-volatile memory of the device.
 11. The method of claim 10,wherein the first capacity is configured to buffer the first sensor datagenerated by the sensors up to a first period of time; and the secondcapacity is configured to buffer the second sensor data generated by thesensors up to a second period of time that is longer than the firstperiod of time.
 12. The method of claim 11, further comprising:operating, by an advanced driver assistance system, the vehicle on theroad based on the sensor data stream, wherein the sensors include adigital camera, a radar, a lidar, or an ultrasound sonar, or anycombination thereof.
 13. The method of claim 12, wherein the signal ofthe first type is from the advanced driver assistance system to indicatean accident involving the vehicle; and the signal of the second type isfrom the advanced driver assistance system to indicate an opportunity totrain the advanced driver assistance system.
 14. The method of claim 13,wherein the advanced driver assistance system includes an artificialneural network and is configured to generate the signal of the secondtype in response to a mismatch between a first identification of anobject determined by the artificial neural network using the sensor datastream and a second identification of the object determined withoutusing the sensor data stream.
 15. The method of claim 13, wherein thefirst sensor data is entirely generated by the sensors prior to thesignal of the first type; and the second sensor data includes a firstportion that is generated by the sensors prior to the signal of thesecond type and a second portion that is generated by the sensors duringa predetermined period of time after the signal of the second type. 16.The method of claim 10, further comprising: stopping, in response to thesignal of the first type, buffering data from the sensor data streaminto the first cyclic buffer before at least a portion of the firstsensor data is copied from the first cyclic buffer into the non-volatilememory; and continuing, for a predetermined period of time and inresponse to the signal of the second type, buffering data from thesensor data stream into the second cyclic buffer before the controllerstops buffering further data from the sensor data stream into the secondcyclic buffer.
 17. A non-transitory computer storage medium storinginstructions which, when executed in a device, cause the device toperform a method, comprising: receiving, via an interface of the device,a sensor data stream from sensors configured on a vehicle duringautonomous operations of the vehicle on a road, the interface operableto receive signals of a first type and a second type; in absence of asignal of the first type and a signal of the second type: buffering, bya first cyclic buffer having a first capacity, the sensor data stream;and buffering, by a second cyclic buffer having a second capacity largerthan the first capacity, the sensor data stream in parallel with thefirst cyclic buffer; copying, by a controller of the device and inresponse to the signal of the first type, first sensor data from thefirst cyclic buffer into a non-volatile memory of the device; andcopying, by the controller of the device and in response to the signalof the second type, second sensor data from the second cyclic bufferinto the non-volatile memory of the device.
 18. The non-transitorycomputer storage medium of claim 17, wherein the method furthercomprises: stopping, in response to the signal of the first type,buffering data from the sensor data stream into the first cyclic bufferbefore at least a portion of the first sensor data is copied from thefirst cyclic buffer into the non-volatile memory; and continuing, for apredetermined period of time and in response to the signal of the secondtype, buffering data from the sensor data stream into the second cyclicbuffer before the controller stops buffering further data from thesensor data stream into the second cyclic buffer.
 19. The non-transitorycomputer storage medium of claim 17, wherein the method furthercomprises: operating, by an advanced driver assistance system, thevehicle on the road based on the sensor data stream, wherein the sensorsinclude a digital camera, a radar, a lidar, or an ultrasound sonar, orany combination thereof; wherein the signal of the first type is fromthe advanced driver assistance system to indicate an accident involvingthe vehicle; and the signal of the second type is from the advanceddriver assistance system to indicate an opportunity to train theadvanced driver assistance system.
 20. The non-transitory computerstorage medium of claim 17, wherein the first sensor data is entirelygenerated by the sensors prior to the signal of the first type; and thesecond sensor data includes a first portion that is generated by thesensors prior to the signal of the second type and a second portion thatis generated by the sensors during a predetermined period of time afterthe signal of the second type.